---
title: "Data Flow — Integration Pipelines"
type: concept
created: 2026-04-18
updated: 2026-04-18
sources: ["raw/articles/00-overview.md", "raw/articles/02-api-contracts.md", "raw/articles/06-reading-telemetry.md"]
tags: [architecture, data-flow, pipelines, integration]
---

# Data Flow — Integration Pipelines

The Pickatale platform has five major integration pipelines that connect the services. All cross-service calls use REST with `X-Internal-Key`. No direct DB access between services.

## Pipeline 1: Reader → Telemetry → Learner Bot

The most critical pipeline — every reading session flows through this.

```mermaid
sequenceDiagram
    participant C as Child (Reader App)
    participant R as Reader App :3125
    participant T as Telemetry :3110
    participant B as Learner Bot :3120

    C->>R: book_opened event
    R->>T: POST /api/events {book_opened}
    T->>T: INSERT events (dedup by event_id)

    loop Per page turn
        C->>R: page_turned event
        R->>T: POST /api/events {page_turned, time_on_page_ms}
        C->>R: word_tapped event (optional)
        R->>T: POST /api/events {word_tapped, word}
    end

    C->>R: session_ended event
    R->>T: POST /api/session/end
    T->>T: Aggregate session (avg_time, slow_pages, word_taps)
    T-->>B: POST /api/v1/telemetry/vocab-taps (fire-and-forget, 5s)
    T-->>B: POST /api/v1/telemetry/session-summary (fire-and-forget, 5s)
    B->>B: Store reading_pattern + engagement_signal memories
```

**Offline mode:** Client queues events in IndexedDB via Service Worker. Auto-replay on reconnect (72h max).

## Pipeline 2: Quiz Result → Learner Bot

```mermaid
sequenceDiagram
    participant C as Child (Reader App)
    participant R as Reader App :3125
    participant AB as Assessment Bot
    participant B as Learner Bot :3120

    C->>R: Post-book quiz submitted
    R->>AB: POST /api/v1/assess/result {answers}
    AB->>AB: Score: <65% → drop, 65-85% → hold, >85% → raise
    AB->>B: POST /api/v1/bot/:learner_id/quiz-result {scorePct, levelRecommendation}
    B->>B: Store assessment_result memory
    B->>B: setImmediate(() => runBotForLearner)
```

## Pipeline 3: Nightly Bot Cycle

```mermaid
sequenceDiagram
    participant CRON as Cron 04:00 UTC
    participant B as Learner Bot :3120
    participant CM as Curriculum Mapper :3100
    participant A as Adaptive Engine :3119
    participant TP as Teacher Portal :3116

    CRON->>B: Trigger nightly run
    loop Per active learner (full tier only)
        B->>B: Query memories, vocab_gaps, assessment_results
        B->>CM: GET /api/v1/objectives (territory, year_level)
        B->>B: Select 1-3 curriculum gaps (ranked by confidence + recency)
        B->>A: POST /api/v1/adapt (book_id, target_fk_grade)
        B->>B: Generate teacher_report (curriculum language)
        B->>B: Generate parent_digest (warm tone)
        B->>B: INSERT nightly_reports
        B->>TP: Insert teacher_notifications if: low_score OR inactive
    end
```

## Pipeline 4: Entitlement Check

```mermaid
sequenceDiagram
    participant S as Any Service
    participant AC as Account Center :3126

    S->>S: Check in-process cache (TTL 60s)
    alt Cache miss
        S->>AC: GET /api/billing/entitlement?school_id=N (X-Internal-Key, 2s timeout)
        AC->>AC: Check entitlement_grants → then subscriptions
        AC-->>S: {tier: 'full'|'free', expires_at}
        S->>S: Store in cache (60s TTL)
    end

    alt Account Center timeout
        S->>S: Fallback to free tier (fail-open)
        S->>S: INSERT audit_log (entitlement_check_failed)
    end
```

**Cache invalidation:** On Stripe webhook → Account Center broadcasts `POST /api/internal/invalidate-entitlement` (fire-and-forget) to all services.

## Pipeline 5: Reader → Adaptive Engine (FK Leveling)

```mermaid
sequenceDiagram
    participant C as Child (Reader App)
    participant R as Reader App :3125
    participant A as Adaptive Engine :3119

    C->>R: Request page N
    R->>A: POST /api/v1/level-page {bookId, pageNumber, text, targetFkLevel}
    A->>A: Check cache (book_id + page_number + rounded_level + text_hash)
    alt Cache hit
        A-->>R: {leveledText, actualFkScore, cached: true}
    else Cache miss
        A->>A: GPT-4o: re-level text to targetFkLevel
        A->>A: Verify |actualFkScore - targetFkLevel| ≤ 0.8 (retry if not)
        A->>A: Cache result
        A-->>R: {leveledText, actualFkScore, cached: false}
    end
    R-->>C: Render leveled page
```

**ZPD offset:** targetFkLevel = child's reading_level + 0.5 (slightly above current level).

**Known issue:** FK scores sometimes unchanged despite GPT call. GPT prompt tightening required.

## Event Types Summary

| Event | Producer | Consumer | Method |
|---|---|---|---|
| page_turned, word_tapped, session_ended | Reader App (client) | Telemetry | Fire-and-forget HTTP POST |
| vocab-taps, session-summary | Telemetry | Learner Bot | HTTP POST, 5s timeout |
| quiz-result | Reader App | Learner Bot | HTTP POST, 5s timeout |
| nightly-bot-run | Cron 04:00 UTC | Learner Bot | Internal trigger |
| entitlement-invalidate | Account Center | All services | HTTP POST broadcast |
| LRS xAPI statement | Reader App | LRS | On session_ended (compliance copy) |